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Knowledge  Discovery  in  Databases  is  the  process  of  extracting  new  patterns  from  ex- 
isting data.  Decision  Tree  Induction  is  the  process  of  creating  decision  trees  from  samples 
of  data  and  validating  them  for  the  whole  data  base.  The  approach  taken  in  this  project 

Several  performance  problems  need  to  be  addressed  for  using  a decision  tree  approach  to 


CHAPTER  i 
INTRODUCTION 


Knowledge  Discovery  in  the  context  of  large  databases  is  an  area  of  growing  interest 
[35]  [12]  [24]  [1]  [3]  [20]  [37]  (2|  [21].  Knowledge  Discovery  or  Data  Mining  is  the  process 
of  making  explicit  patterns  that  are  implicit  in  the  dalai  been  analysed.  These  patterns 
represent  knowledge  embedded  in  the  data  under  consideration.  Discovering  them,  i.e., 
making  them  explicit,  is  the  subject  of  every  data  mining  system. 

However,  there  is  not  general  agreement  on  which  type  of  patterns  must  be  discovered; 

whole  data,  such  as  “if  the  temperature  is  higher  then  100  degrees  then  color  is  red”;  “if 
the  customer  spends  more  than  $50  then  a gift  is  given"  and  “if  region  is  northwest  then 
precipitation  is  high".  Additionally,  to  bo  of  practical  interest  it  is  important  to  know  the 

Data  mining  requires  the  convergence  of  several  fields:  data  bases,  statistics,  machine 
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Figure  1.1.  Knowledge  Discovery  Model 


Database  Interface:  Mosl  recent  work  on  data  mining  can  be  called  file  mining  [22] 
because  it  lacks  this  primary  component:  a way  to  access  existent  databases  by  an 
interface  language.  See,  for  example,  Han  et  al.  [21]. 


processing  the  entire  data  set  ( which  is  typically  very  large). 

Pattern  Extraction:  This  part  is  the  specific  way  to  extract,  manipulate  and  represent 
specific  patterns  from  the  database.  A mechanism  able  to  search  for  specific  patterns 
like  if-then  rules,  semantic  nets  or  decision  trees. 


knowledge  database  (a  data  base  of  rules  and  domain  knowledge  -in  form  of  rules-  or 


with  the  i 


Common  to  all  data  mining  systems  are  two  primary  functions:  classification  and  rule 
representation.  Classification  is  useful  as  a way  to  group  data  and  focus  the  data  analysis 


algorithms  have  been  proved  to  be  good  classifiers  in  the  machine  learning  field.  Induction 
its  lach  of  application  to  large  data  bases, 

Their  use  in  inductive  inference  based  systems  for  small  data  sets  has  been  very  well 
investigated  and  documented.  In  addition  to  their  ability  to  classify  new  data,  decision 


• They  can  represent 


A decision  tree  derived  from  the  data  can  capture  potential  rules  present  in  the  data. 


decision  tree  associated  with  (or  derived  from)  this  partition  determines  the  respective 


i reduce  its  sine,  improve  its  dassi- 


I have  found  that  decision  trees  can  function  correctly  and  efficiently  only  if  we  provide 
those  functions  and  capabilities  described  for  a Knowledge  Discovery  model  to  any  decision 
tree  based  system. 


in  knowledge  discovery;  bu 

The  use  of  decision  trees  in  very  large  databases  and  in  distributed  ones  requires  a 
compromise  so  they  can  operate  efficiently  in  such  an  environment  while  preserving  the 
accuracy  and  quality  of  the  knowledge  discovered.  There  is  also  a need  for  designing  a 
suitable  interface  and  the  data  mining  operators  needed  to  retrieve  data  from  the  database. 
However,  my  concern  is  with  classification  and  rule  representation  with  decision  trees  in 


CHAPTER  2 

SYSTEMS  FOR  KNOWLEDGE  DISCOVERY  IN  DATABASES 


SLIQ  was  developed  by  M.  Mehta,  R.  Agrawal  ati  .1.  Rissanen  at  the  IBM’s  Alniaden 
Research  Center  [26].  The  objective  of  SLIQ  is  to  solve  the  classification  problem  for  data 
mining  using  scalable  techniques.  It  is  a decision  tree  extraction  system  for  very  large  data 
sets  that  creates  inverted  lists  for  the  attributes  and  uses  splitting  /subsetting  of  attributes 


Class  list:  An  ordered  list  of  class  values  for  each  tuple  and  the  corresponding  decision  node 
path  of  nodes  from  the  root  to  the  leaf  node.  Initially  all  tuples  are  associated  with 


; tree.  Initially,  inverted  lists  for  each  attribute  are  created.  Then, 


attribute  selection  is  done  by  pre-sorting  numerical  attributes  and  by  finding  the  best 


Decision  Tice:  A binary  decision  tree.  In  each  node,  it  keeps  the  node  number,  the  decision 
attribute,  the  decision  value  and  the  class  histogram.  ( Class  counts  for  every  class 


value  to  the  left  and  right  of  the  decision  value). 

Sliq  (): 


SO-  Rend  database  and  create  a separated  list  for 
each  attribute  and  tuple  index  (Attribute  list). 
For  every  class  value,  associate  an  initial  node 
nl  (the  leaf)  and  create  the  list  with  class  values 
and  nodes.  (Class  List). 

Sort  all  attribute  lists  by  attribute  value. 

S2-  Partitionfdata  S): 

52.0-  If  (all  tuples  are  in  the  same  class)  return; 

52.1-  Evaluate  Splits: 


For  each  attribute  A do 

Update  the  class  histogram 

if  A Is  a categorical  attribute 

Find  subset  of  A with  best  split 


For  each  value  v do 

-Find  the  entry  in  the  class  list,  e. 

-Find  the  new  class  c to  which  v belongs 
by  applying  the  splitting  test  in  node  referenced  by  e. 
-Update  the  class  label  for  e to  c 
-Update  the  node  referenced  in  e to  the  child 


Update  class  list. 


52.4-  Partition  (SI) 

52.5-  Partition(S2) 

2.1.2  M frits 

Sliq  performs  similar  to  or  better  than  others  classifiers  for  small  data  sets  and  the 
classification  tune  is  almost  linear  for  large  data  sets  [26].  This  is  the  first  case  where  more 
than  100000  instances  were  used  (until  10  million).  The  pruning  method  influences  signifi- 
cantly its  performance.  Important  contributions  are  scalability  and  breadth-first  growth  as 
well  as  subsetting  for  categorical  attributes  and  pruning  using  the  Minimum  Description 
Length  principle.  The  use  of  synthetic  databases  with  more  than  100000  cases  prove  the 
scalability  of  SLIQ. 


Sliq  derives  a decision  tree  that  correctly  classifies  the  training  set  and  gets  high  accuracy 
for  the  whole  set,  but  it  is  not  incremental.  It  is  designed  to  classify  the  training  data  set 
but  without  using  induction  or  learning  capabilities  te.,  it  processes  the  entire  database  to 

SLIQ  makes  at  most  two  complete  passes  over  the  data  for  each  level  of  the  decision  tree  [26, 

splits  for  each  attribute,  making  this  phase  a time  consuming  task. 

2.1.1  Summary  of  SLIQ  Features 


l.  When  symbolical  or  categorical  attributes  (strings)  are  used,  the  amount  of  space 


required  is  increased  by  Ihc  sine  of  the  indexes  to  the  data  base.  The  algorithm  is  faster  in 
the  sense  that  it  uses  just  one  pass  for  every  level  of  the  decision  tree,  but  the  actual  volume 
of  the  inverted  lists  is  almost  two  times  the  initial  data  base  volume,  increasing  the  number 
of  I/O  accesses.  This  is  particularly  significant  in  a very  large  data  base  environment. 


Several  algorithms  have  been  proposed  to  extract  association  rules  from  data  |1),  [19), 
[3],  [37],  (20),  [2].  Most  of  these  systems  are  based  on  the  original  algorithm  proposed  by 

2.2.2  The  Aoriori  Algorithm 

The  basic  algorithm  is  summarized  below: 

The  Apriori  algorithm: 
si  L(l)  = frequent  ldtomsets 

s2  k = 2;  //  k is  the  pass  number  (<=  number  of  attributes! columns)) 
s3  while  ( L(h-l)  is  not  empty  ) do 

si  C(h)  = New  candidates  of  size  k generated  from  L(k-l); 

s6  For  all  c(k)  in  C(k)  do  c(k).count++; 
s7  L(k)=  all  candidates  in  C(k)  with  minimum  support; 


slO  Answers  Union  of  L(k),  for  all  k. 


2.2  Systems  that  Extract  Ulrica  from  Pats 


if  A is  the  number  of  attributes,  then  Stop  s3  is  clone  A times  in  the  worst  case.  In  step 


s5,  the  whole  data  base  is  traversed.  So,  we  have,  at  most,  A number  of  passes  over  the 
data  base. 

data  base  and  therefore  the  number  of  I/O’s  incurred  for  that  purpose. 

An  improvement  to  the  previous  algorithm,  called  the  AprioriTid  algorithm  proposed 
also  proposed  by  Agrawal  and  Srikant  (31,  suggested  that  a data  structure  be  used  to  discard 


The  goal  of  Parallel  systems  is  to  extract  association  rules  by  using  parallel  processing 


Approach: 

This  is  achieved  by  parallelising  the  serial  algorithm  -the  Apriori  algorithm-  which 
counts  the  support  of  each  itemsets  and  finds  rules  based  on  the  frequent  itemsets.  The 


l.  The  count  distribution  algorithm  - in  which  basically  each  processor  counts  the 


2.  The  data  distribution  algorithm  in  which  the  total  memory  of  the  system  is  exploited 


10 

exclusive  candidates  (viable  itemsets).  and  then  the  local  data  must  be  broadcast  to 


3.  The  last  algorithm  (the  candidate  distribution ) tries  to  make  each  processor  work 
independently  since  in  the  previous  algorithms  each  processor  locally  extracts  the 
candidate  sets.  Synchronization  is  needed  at  the  end  of  every  pass.  The  idea  is 

sors  dividing  appropriately  the  frequent  itemsets.  However,  not  all  dependencies  are 
eliminated. 


Additionally,  a parallel  algorithm  is  presented  to  generate  roles  from  frequent  itemsets. 
Merits  of  the  three  approaches: 

The  three  algorithms  give  dear  ideas  of  how  to  parallelize  the  serial  algorithm.  The 
Limitations: 

-where  databases  were  increased  proportionally  to  the  number  of  processors,  -the  Count 

speedup,  keeping  the  database  constant  and  adding  more 
tion  is  better  and  performs  almost  linear  up 


The  number  of  paeses  over  the  data  is  the  same  for  all  algorithms  except  the  data 
distribution  algorithm.  The  number  of  passes  is  proportional  to  the  transaction  length 
(since  those  are  binary  values  and  each  represents  an  attribute-value,  we  may  say  that  the 
passes  are  proportional  to  the  number  of  attributes  in  the  relation). 

In  the  above  approaches  the  whole  database  is  processed  as  no  learning  algorithms  arc 

2.2.4  The  Partition  Algorithm  for  Deriving  Association  Rule; 

Another  algorithm  called  the  Partition  algorithm,  introduced  by  Sarasore  ct  al.  [37], 


Basically,  the  algorithm  avoids  passing  over  the  data  in  step  s5  of  the  Apriori  algorithm. 
Instead  of  reading  the  data  base  again,  to  count  the  support  of  the  Candidate  sets,  it  keeps 
the  transaction  list  of  each  set.  Counting  is  done  by  taking  the  intersection  of  those  lists. 

The  algorithm  is  called  Partition,  since  it  can  apply  the  modified  Apriori  algorithm  to 
parts  of  the  database ; then  all  local  large  itemsets  are  joined  to  get  the  final  large  itemsets. 
In  order  to  merge  all  local  large  itemsets,  an  additional  pass  is  necessary. 

The  performance  results  show  that  for  lower  minimum  support  values  (less  than  1%)  the 
Partition  algorithm  outperforms  the  Apriori  Algorithm.  The  reason  for  this  (our  opinion)  is 
that  lowering  the  support,  the  transaction  lists  of  each  itemset  are  shorter  and  can  be  kept 


small  values  of  support. 


CHAPTER  3 

DECISION  TREE  CONSTRUCTION 


3J The  Tree  Construction  Algorithms 

ee  induction  was  introduced  by  J . R.  Quinlan  [27]  [31]. 

and  UtgolT[38],  [44].  Those  algorithms  requires  one  pass  over  previously  seen  data  per  level 

data,  either  directly  or  incrementally. 

3.2  The  Centralized  Decision  Tree  Induction  Algorithm 

ee  induction  [27,  pp  469]  was  as  follows: 


(s2)  Repeat 


(S2.2) 

(.2.3) 


Find  the  exceptions  of  this  decision 

window  plus  the  exceptions  to  the 
decision  tree  generated  from  it 
itil  there  are  no  exceptions 


Step  2.1  is  called  Decision  Tree  Derivation  and  step  2.2  is  called  Decision  TVee  Testing. 


Step  2.3  is  the  major  drawback  in  the  above  algorithm  since  it  forces  the  process  to  pass  over 
all  the  training  data  (the  window)  again,  and  therefore  the  algorithm  is  not  incremental. 
The  algorithm  presumes  that  none  of  the  instances  are  stored  within  the  decision  tree 

information  is  needed  in  each  node  besides  the  decision  data. 


The  Decision  Thee  Derivation  (step  2.1)  proceeds  in 
followed  by  a partition  stage: 

(82.1.1)  Select  the  best  attribute  (the  root) 


selection  stage 


Slops  82.1.1  and  >2.1.2  of  this  algorithm,  the  selection  and  partition  steps,  respectively, 
each  require  one  pass  over  the  data  set.  Selection  steps  usually  count  the  relative  frequency 
in  the  data  set  of  every  altribule*value  with  the  class  value  (Class  counts)  which  are  then 
used  statistically  to  compute  the  host  attribute  (the  root).  The  partition  steps  distribute 
the  data  across  the  different  branches  of  the  root  attribute.  Thus,  the  algorithm  in  general 
requires  two  passes  over  the  data  per  level  of  the  decision  tree  in  the  worst  case. 

3.?  The  Selection  Criteria 

suggested  by  Quinlan  [27).  The  information  gain  criterion  minimiles  the  average  attribute 

£M)=  £ PtA=a)tl„(A  = a)  (3.1) 

where  P(A=a)  is  the  relative  probability  of  A = a,  and  for  a set  of  n potential  classes, 

/7„(d  = e)  = -£  p,(,t  = e)log(p,(d  = o))  (3.2) 

mizing  the  entropy,  maximizes  the  certainty  and  is  given  by: 


£(/l)  = 53  P(/l  = a)CH„(A  = i 


(3.3) 


ifrjx 


(sO)  Select  a random  subset  of  the  data  base  (the  window) 
(si)  Build  the  decision  tree  to  explain  the 

current  window  (the  Tree)  -keep  Class 
counts  in  every  node. 

(s3)  While  there  are  exceptions;  do 


Incremental  algorithms  usually  start  with  a random  subset  of  one  element.  The  algo- 
rithm above  doesn't  preclude  this  possibility. 


Tree  reorganization  is  the  key  for  incremental  algorithms  including  algorithms  which 
are  not  based  on  statistics  over  the  input  instances  [10].  This  technique  is  essential  to  avoid 
traversing  the  whole  data  base  again  when  dealing  with  very  large  databases.  Hopefully,  tree 
reorganization  will  require  just  a small  part  of  the  database  when  the  tree  is  restructured. 

The  reorganization  part  depends  on  the  relative  representation  suited  for  the  algorithm. 
Utgoff  maps  every  altribulc-value  pair  to  a new  boolean  attribute  [45].  Thus,  he  assumes 

Tree  reorganization  algorithms  restructure  the  tree  when  a better  attribute  is  detected 


(s3.2) 


Find  a exception  of  tbe  decision  tree 
in  the  remaining  instances. 

Update  the  decision  tree  Class  counts 


’ node  using  this  exception. 


(•3.3) 


Reorganize  (Tree);  See  below. 


The  ID5R  pull  up  algorithm  reorganizes  the  decision  tree  in  the  way  just  mentioned 
[44].  If  a tree  is  just  a leaf  ( a set  of  instances) . the  pull  up  algorithm  assumes  the  respective 


The  ID5R  pull  up  algorithm 


Transpose  the  tree,  resulting  in  a new  tree  with  A at 
at  the  root,  and  the  old  root  attribute  at  the  root 
of  each  immediate  subtree. 


Note  that  in  step  s2.2  the  transformation  rules  of  figure  3.3  must  be  applied  to  obtain 

Van  de  Velde 's  algorithm  IDL  uses  the  same  pull  up  technique  for  reorganization  as 
ID5R  (10].  IDL  differs  from  others  in  that  it  uses  a topological  criterion,  called  topological 
relevance  gain,  based  on  the  tree  structure  to  select  the  actual  root  attributes  for  the 
subtrees.  Van  de  Velde  shows  how  his  algorithm  is  ab 
shown  in  figure  3.4;  while  traditional  algorithms  fail  to  discover  this  In 


Figure  3.4.  A tree  for  the  6-multiplexer 
currences  of  a given  attribute  A for  a given  example  c when  this  is  used  to  traverse  the  tree 

m and  s in  the  classification  path  of  an  example  e,  with  s the  immediate  son  of  m,  the 


TRG„(. 


TRm(A,e) 


(3.5) 


When  compared  to  its  predecessor  ID5R,  1DL  saves  computations  co 
keeps  better  or  similar  accuracy. 

More  recently.  Utgoff  has  implemented  the  ITI  algorithm  which  is  a direct  descendant 
of  ID5R  and  uses  reorganization-like  techniques  in  a similar  way  [45). 

3,5  Other  Approaches 

SUQ  , a fast  scalable  classifier  for  Data  Mining  (described  in  chapter  2,  was  designed 
to  solve  the  classification  problem  for  Knowledge  Discovery  [26].  Conceptually,  the  SL1Q 
system  uses  the  same  algorithm,  where  the  selection  criterion  is  the  gini  index  - a criterion 


G(S)  = 1 (3.6) 

G(A  = a)  = P( A < a) . G(4  < n)  + /•(/!  > a)  * G( A > a)  (3.7) 


Thus,  SUQ  representation  is  a binary  decision  tree.  In  order  to  make  the  system  scal- 
able, most  of  the  data  are  handled  offline  with  inverted  lists  for  all  attributes  and  a special 
Class  list  that  maps  the  instances  to  the  nodes  of  the  decision  tree.  This  Class  list  is 
maintained  in  main  memory.  The  system  incorporates  tree  pruning  using  the  Minimum 
Description  length  principle  . Mehta  et  al  show  that  the  system  achieves  similar  or  better 
performance  than  IND-Cart  and  IND-C4  (ID3  descendants)  for  different  data  sets;  espe- 
cially for  larger  data  sets  (20000  to  60000).  They  also  show  that  for  synthetic  data  bases 
of  millions  of  cases,  SLIQ  achieves  almost  linear  performance  on  the  number  of  tuples  and 


Decision  trees  for  Knowledge  Discovery  in  large  databases  can  be  applied  in  two  related 
areas:  classification  and  rule  extraction.  Although,  rule  extraction  from  decision  trees  is  not 

Nevertheless,  the  decision  tree  algorithms  mentioned  have  the  following  problems  when 
used  for  large  databases: 


1.  Their  study  has  been  primarily  done  on  small  data  sets  (from  hundreds  of  cases  to  a 
to  large  data  sets.  See  [26]. 


me-  will  preclude  its  use  over  a direct  derivation  algorithm  over 


There  has  not  been  any  related  work  on  the  mapping  between  decision  trees  and  asso- 
ciation rules  for  data  mining,  bevels  of  support  and  confidence  in  decision  trees  and 
its  relationship  to  the  attribute  selection  criterion  is  not  mentioned  in  the  references. 
Traditional  algorithms  assume  that  instances  and  class  counts  are  kept  in  memory 
regardless  of  the  number  of  tuples  involved.  Sliq  is  the  other  extreme  case  where  all 

Recent  algorithms  like  1TI  and  Sliq  represent  or  transform  the  attributes  to  binary 
attributes  is  a must  for  the  end  user. 


Induction  techniques  for  large  or  distributed  databases  have  not  been  studied. 


CHAPTER  4 

EXTENSIONS  OF  DECISION  TREE  CONSTRUCTION  ALGORITHMS 


The  basic  algorithm  for  decision  tree  induction  introduced  by  J.  R.  Quinlan  had  two 
major  drawbacks  for  its  use  in  very  large  databases:  it  was  not  incremental  and  it  required, 

The  incremental  solution  based  on  tree  restructuring  techniques  (38]  [44]  requires  one 
pass  over  previously  seen  data  per  level  in  the  worst  case,  as  does  Van  de  Velde's  incremental 

incremental  version  more  attractive  for  large  databases.  However,  the  incremental  version 
requires  keeping  the  data  “inside”  the  decision  tree  structure  [45]  in  main  memory  and 

large  databases.  This  section  describes  a one-pass  per  level  worst  case  algorithm  to  build 

databases  requires  a mechanism  to  store  part  of  the  tree  in  externa]  memory.  Since  the  size 
of  large  databases  precludes  keeping  several  copies  of  the  data,  data  can  be  incorporated 

the  database. 


’ the  data  base,  the  split  step  and  the  selectia 


of  the  next  step  need  to  he  combined  in  one  pass.  The  trick  is  to  use  each  case  (tuple) 
to  update  the  Class  counts  of  the  corresponding  subtree  (or  subset)  and  to  create  the 
data  subset  simultaneously.  Thus,  in  the  next  selection  step,  there  will  be  no  need  for  an 
additional  pass  over  the  subsets  for  every  subtree  in  the  next  level.  Then,  even  in  the  worst 
case,  we  will  need  only  one  pass  per  level  over  the  data  base. 


The  first  step  of  the  derivation  must  proceed  like  this: 


Derivation  Revisited  (Initial  step) 


(&2.1.0)  If  all  instances  are  of  the  same  class, 

the  tree  is  a leaf  with  value  equal  to  the 
class,  so  no  further  passes  are  required. 


(&2.1.2)  Split  the  set  of  instances  according  to  each 
value  of  the  root  attribute. 

Update  Class  Counts  for  every  subtree  with  each  instance 


(s2.1.3)  Derive  the  decision  subtree  for  each  subset 
of  instances. 


(s2.1.0)  If  all  instances  are  of  the  same  class, 

the  tree  is  a leaf  with  value  equal  to  the 
class,  so  no  further  passes  are  required. 
(s2.1.1)  Get  the  best  attribute  (the  root) 
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Update  Class  Counts  for  every  subtree  with  each  instance 
(s2.1.3)  Derive  the  decision  subtree  for  each  subset 


Note  that  the  initial  step  requires  two  passes  to  check  the  data.  After  that,  the  remaining 
steps  require  just  one  pass  per  level.  The  selection  step  does  not  require  a pass  over  the 


The  merging  of  these  two  steps  is  not  without  cost.  Additional  memory  is  required 
to  keep  all  frequencies  (Class  counts)  for  every  subtree.  If  we  keep  all  those  frequencies 
in  memory,  then  it  is  clear  that  the  decision  tree  can  be  built  for  every  subtree,  without 
additional  disk  accesses.  Note  that  only  the  class  counts  for  the  last  level  are  needed  and 
that  the  number  of  counts  maintained  in  main  memory  are  fewer  whenever  the  level  (of 
the  tree)  is  higher.  However,  there  can  be  thousands  of  leaves  in  a decision  tree  for  a large 
data  base.  Eventually,  a mechanism  to  keep  the  class  counts  outside  of  main  memory  is 


can  easily  be  implemented. 
1.2,2  The  Selection  Criteria 


data  since  all  Class  Counts  were  computed  previously. 


e.  The  number  of  additional  disk  accesses  for  reading  class 


. criterion  lias  several  limitations  [31),  I am  proposing  an  alter* 


where  pj(A  = a)  = maxiPi{A  = «) 

Then,  the  average  certainty  per  attribute  is  given  by: 


£<A)  = £ P{A  = a)Dn(A  = a) 


Intuitively,  the  determination  guesses  the  most  probable  class  in  a given  data  set  based 
uniquely  on  their  relative  probabilities.  See  chapter  5 for  more  details  on  the  criterion. 

Both  measures,  certainty-based  entropy  as  described  in  chapter  3 and  the  determination 
- being  the  base  for  tree  derivation  - allow  us  to  study  the  inductive  behavior  of  the  decision 


The  Tree  Derivation  Algorithm  halts  when  all  instances  in  the  data  set  are  from  the 
same  class  (step  2.1.0).  It  is  impractical  to  expect  this  since  data  can  be  inconsistent  or 

Thus,  a threshold  criterion  must  be  introduced  to 

and  select  attributes  in  step  2.1.1. 

positives  and  5%  negatives.  The  best  selected  attribute  will  be  either  of  them;  but  the 
average  measure  will  be  the  same  since  the  relative  distribution  of  classes  in  each  leaf  is 


Table  4.1.  Entropy  conjecture 


0.47  entropy  (0.53  certainty)  in  both  cases.  As  the  result  is  equal  to  the  set  measure,  no 
improvement  has  been  made. 

Even  though,  the  previous  example  is  an  extreme  case,  usually  absent  in  practice,  the 
algorithm  must  check  for  this  condition.  In  general,  the  algorithm  must  check  if  the  average 
certainty  -equation  4.2-  is  below  or  equal  to  the  set  certainty. 

For  the  entropy  measure,  the  following  seems  to  be  true: 

Confecting  f Let  S the  data  set.  A an  attribute.  Then 


E(A)=  53  P(A=a)CH„{A  = a)> 


This  says  that  the  entropy-based  certainty  will  always  be  greater  or  equal  to  the  set 
certainty  higher  than  the  original  set  entropy-based  certainty  (column  1). 
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For  the  Determination  measure,  the  conjecture  is  not  true.  For  example,  with  90% 
positive  cases  and  10%  negative  cases,  ( 0.88  Determination),  the  partition  in  one  set  of 
80%  positive  and  0%  negative,  and  another  of  10%  positive  and  10%  negative,  does  not 
lead  to  a better  average  determination  (0.8(1)  + 0.2(0)  = 0.80).  Note  that  the  entropy 
(certainty)  changes  from  0.47  (0.53)  to  0.20  (0.80). 

This  property  of  the  Determination  measure  will  allow  us  to  prune  the  decision  tree 
before  it  fits  the  data  unnecessarily  since  there  is  no  improvement  in  the  measure.  On 

the  classification)  since  entropy  decreases  (certainty  increases)  with  every  partition  if  the 


rasures  is  shown  in  figure  3.2.  Note  that  the  certainty  always  increases  when  entropy 

rived  tree  will  be  the  tree  depicted  in  figure  4.2.  Note  that  both  subtrees  starting  with 
ot  Symptom  B were  not  needed  since  E(B)  was  always  lower  than  the  respective  set 


determination  (del).  This  coincides  with  the  fact  that  the  determination  chooses  the  most 
general  rule-  See  chapter  5. 


to  prune  the  tree.  The  most  general  method  is  called  the  Minimum  Description  Length 
principle  introduced  by  Quinlan  [33).  It  has  been  succcsfully  used  in  most  of  the  actual 
systems  [45],  [25],  [26]  . However,  it  improves  the  accuracy  and  reduces  the  size  of  the 

the  pruning  is  artificial  and  of  little  or  not  interest  to  the  user  and  the  application, 
the  meaning  of  the  rules  to  be  extracted  as  criteria  to  prune  the  tree.  The  user  can  specify 

the  final  error  or  the  amount  of  work  needed  to  build  the  tree.  Our  goal  is  to  meet  the 


Symniom  A (0.45,0.55) 


Figure  4.2.  Pruned  decision  tree  with  determination 


but  applicable  in  a different  way  is  the  mechanism 


confidence  and  support  thresholds. 


Table  4.2.  Pruning  with  tin.  d<  lermin  itio 


Similarly,  the  attribute  selection  criterion  gave  us  a good  tool  Tor  tree  pruning  if  we 

for  this,  since  there  is  no  way  to  relate  the  entropy  measure  to  the  set  confidence,  fn  my 
opinion,  this  is  the  primary  reason  for  the  developing  of  pruning  criteria  such  as  the  MDL 


without  sacrificing  largely  the  error  rate  (no  more  than  5%). 

■1.3  Extensions  to  the  Incremental  Algorithms 
As  mentioned  in  chapter  3,  the  incremental  algorithm,  originally  devised  by  Utgoff  [3-1), 
avoids  passing  unnecessarily  over  previously  seen  instances. 
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With  our  one  pans  algorithm,  it  is  necessary  to  re-evaiuale  the  incremental  version  - 
since  the  cost  of  both  approaches  is  O(n)  in  general.  However,  direct  tree  derivation  is  a 
pessimistic  approach  and  assumes  Nothing  about  the  data.  Incremental  algorithms  are 
optimistic  and  they  assume  that  the  previous  decision  tree  reflects  the  actual  decision  tree. 

improved  as  compared  to  the  direct  (brute-force)  approach.  I will  discuss  more  thoroughly 
the  re  organisation  approach  used  in  incremental  algorithms  in  a later  section. 

In  genera],  the  cumulative  cost  of  the  pure  incremented  algorithm  -one  instance  at  a 
time-  will  preclude  its  use  over  a direct  derivation  algorithm  over  the  data  base.  The  algo- 

(the  updating  phase)  using  chunks  of  wrongly-classified  instances  instead  of  one  instance 


Partial  Incremental  Induction  Algorithm: 
))  Select  a random  subset  of  the  data  base 
(the  window) 

in  the  remaining  instances. 


(s3.4) 


Reorganize  (Tree);  See  below. 

Find  the  exceptions  to  this  decision 


> in  the  remaining  instances. 


a better  attribute  is  detested  (or  inherited  in  the  cose  of  a subtree).  A more  detailed 
algorithm  for  tree  reorganization  is  given  below.  Again,  I have  based  the  algorithm  on  the 
transformation  rules  in  figure  3.3  of  chapter  3. 


he  new  root  attribute  (NewRoot). 


ReorganizefTree,  NcwRoot): 

(si)  If  the  NewRoot  is  null  then 

(s2)  If  'IYee  is  a leaf, 

(s2.1)  Create  a new  tree  by  splitting  tbe 

(s2.2)  Make  Tree  equal  to  this  new  Tree. 

(S2.3)  return 

(s3)  otherwise  (If  TVee  is  not  a leaf) 

(s3.1)  If  Tree. Root  - — NewRoot  then 


(s3.2.1) 

(s3.2.2) 


Apply  the  transformation  rule 


Update  class  counts  for  the 
subtrees,  (now  starting  with 

Reorganise!  STree) 


decision  tree.  This  is  repeated  for  the  next  level  of  the  decision  tree  until  all  subtrees  hold 
the  best  attributes  as  roots  or  until  they  arc  just  leaves.  There  is  a potential  for  doing  a pass 
over  the  data  at  the  leaves  for  each  level  and  therefore  the  algorithm  requires  one  pass  per 

a subroot  of  a subtree  and  there  is  no  need  to  reorganise  the  subtree.  Thus,  this  algorithm 

the  actual  decision  tree,  which  is  likely  since  the  tree  was  based  on  a representative  subset 


Figure  4.3.  Tree  Reorganization 


Distributed  Induction  of  Decision  Trees 


The  partitioning  part  of  the  Derivation  algorithm  (step  s2)  can  easily  be  adapted  to  a 
multiprocess  or  a multiprocessor  environment.  Every  data  subset  obtained  in  the  partition 
is  given  to  each  available  processor  to  continue  with  the  tree  derivation.  Thus,  the  tree 
induction  mechanism  can  easily  be  made  in  parallel.  Additionally  the  subsets  can  be 
kept  on  secondary  storage  thereby  allowing  even  larger  sets  to  be  used  for  induction  with 
the  restriction  that  the  tree  must  be  loaded  into  memory  if  the  whole  tree  is  needed  for 
processing  (for  example,  for  a centralised  testing  phase).  However,  it  is  possible  to  design 

updating  phase  will  proceed  like  any  centralized  algorithm.  I term  this  the  DSD  algorithm. 

The  DSD  algorithm 
(si)  Make  a pass 

the  attribute  (the  root). 


le,  attaching  to 


(s6)  Exit. 


In  step  s3,  the  relative  speed  of  every  available  processor  can  be  taken  into  account  ■ 


every  subset  will  simply  be  distributed  on  a lirst-come  first-served  basis.  Similarly,  in  order 
to  fully  use  the  distributed  capabilities  of  the  system,  a set  will  be  available  if  its  size  is 
greater  than  a threshold  set  previously  by  the  user. 

The  algorithm  is  useful  when  several  processes  or  processors  can  cooperate  to  help  in  the 
decision  tree  derivation.  It  is  assumed  that  they  at  least  share  a file  system.  For  example, 
the  algorithm  can  be  used  when  the  decision  tree  does  not  fit  in  the  memory  available  for 


An  alternative  use  of  distributed  processing  capability  in  deriving  decision  trees  is  to 

change  class  counts  on  every  attribute-value  pair  and  then  each  one  will  arrive  at  the  same 
conclusion  on  the  selected  attribute  as  a root.  Then,  as  each  data  set  will  be  partitioned 
accordingly,  a new  interchange  of  class  counts  will  occur  for  each  possible  subtree,  until 

minimum  since  data  sets  are  not  interchanged,  just  the  attribute- value-class  frequencies  or 
Class  counts  (See  Figure  4.4  ).  Tills  will  be  called  the  DTD  algorithm. 

The  DTD  algorithm: 

(si) 


Ill  the  above  algorithm,  the  Claes  Counts  interchange  among  processors  can  be  improved 

stage.  This  Coordinator  will  notify  each  processor  the  next  root  at  every  subtree.  Each 

one  copy  of  the  Class  Counts  will  be  transmitted.  With  the  coordinator,  the  number  of 
messages  transmitted  will  change  from  0{n2)  to  0(n),  where  n is  the  number  of  processors. 
The  revised  DTD  algorithm  is  given  below: 


The  Revised  DTD  algorithm: 


Send  the  Class  Counts  to  the  Coordinator. 

processor  and  summarize. 

Select  the  best  attribute  (the  root). 


recursively  derive  the  tree. 


A root  will  be  defined  when  the  message  from  the  Coordinator  is  received  or  when  the 
coordinator  itself  determines  the  root. 


The  update  phase  in  a distributed  setting  is  handled  as  follows.  Each  processor  will 
receive  its  respective  update  data,  get  its  partial  class  counts  and  send  them  over  to  all  other 
processors  or  the  coordinator.  Each  processor  will  receive  all  class  counts  and  update  the 

time  will  be  saved  if  we  use  the  coordinator  approach.  In  this  case,  the  final  tree  must  be 

If  one  were  to  use  any  incremental  algorithm,  such  as  the  IT1  algorithm  [45]  or  the 
algorithm  by  Schlimmer  [38]  and  the  algorithm  is  based  on  class  counts,  then  it  is  possible  to 
employ  the  approach  proposed  here  for  the  derivation  and  updating  every  tree  incrementally 

array  of  frequencies  will  be  at  most  250,000  entries.  The  expected  universal  domain  for  a 
It  is  clear  that  if  a processor  keeps  only  a few  tuples,  it  will  be  better  to  transmit  these 


nt  have  shown  better 


i following  algorithm  derives  the  decision  tree  for  a set  of  attributes  in  the  database. 


It  derives  the  trees  breadth  first  (dilTcrent  from  our  Revisited  Derivation  algorithms)  and 
reads  the  data  base  once  at  each  level  of  ail  trees.  Thus,  we  extract  all  trees  with  A passes 
over  the  database. 

Multiple  Goal  Tree  Derivation  (Initial  step) 

(s2.1.0)  Read  database  and  create  class  counts  for  each  Goal  Attribute. 

(■2.1.1)  For  all  goal  attribute  G do 

the  tree  is  a leaf  with  value  equal  to  the 
Select  the  best  attribute  (the  root) 


Distribute  the  instance  according  to  the 

Update  Class  Counts  for  the  subtree. 
(sR.1.3)  Multiple  Derive  the  decision  subtree  for  each  subset 


Then,  for  each  subtree: 

Multiple  Goal  Tree  Derivation  (MGTD  algorithm) 
(s2. 1.1)  For  all  goal  attribute  G do 
For  all  subset  do 

Get  the  best  attribute  (the  root) 


(82-1.2)  For  each  instance  do 


For  each  Goal  attribute  do 
For  each  subtree  Root  attribute  do 


(s2.1.3)  Multiple  Derive  the  decision  subtree  for  each  subset 


approach  to  solve  this  situation  has  been  either  to  choose  one  option  using  additional 


In  this  chapter,  1 described  how  part  of  the  problems  mentioned  in  chapter  3 can  bo 


s.  Tree  construction  in  this  case  is  not  different  from  the  algorithms 


solved.  Extending  the  algorithms  to  large  data  bases  requires  memory  optimisation,  mini* 
miration  of  I/Os,  and  the  use  of  incremental  approaches.  Distribution,  Parallelism,  Multiple 


Goals  and  Non-determinism  are  neccesary  to  process  massive  amounts  of  data.  The  DTD 
algorithm  can  successfully  obtain  the  decision  tree  in  every  processor  for  a distributed  data 
base.  The  DSD  algorithm  is  useful  in  parallel  machines  or  in  environments  where  file  sys- 
tems are  shared  among  all  processors  (local  area  networks,  clustered  disks).  The  MGTD 
algorithm  is  useful  for  extracting  all  data  dependencies  ( rules)  simultaneosly.  The  fact  that 

those  approaches  very  flexible.  Using  tree  reorganization  for  large  data  bases  looks  expen- 
sive at  first  glance,  but  if  the  tree  has  already  been  derived,  non-pure  incremental  methods 


CHAPTER  5 

THE  DETERMINATION  MEASURE 


In  this  chapter. 


lion  called  the  determination  measure;  I discuss  its 


5.1.1  Fundamentals  of  the  Determination  Measure 

Classification  is  the  mapping  of  objects  to  specific  classes.  In  most  applications,  this 
mapping  is  not  unique  and  an  object  can  be  assigned  to  different  classes.  Thus,  given 

been  used  to  evaluate  the  classification  defined  for  this  probabilities  [30],  [31],  [15],  [16], 
[18],  [33],  |39],  [41].  If  an  object  is  mapped  to  a class  with  high  probability  and  with  low 


most  famous  and  common  measure  is  information  entropy;  since  the  set  of  n potential 
classes  can  be  seen  as  a channel  output  [30].  [39] 


be  considered  a good  one.  Among  others,  the 


(5.1) 


Tims,  a low  entropy  value  is  interpreted  as  a minimum  amount  of  uncertainty  (high 
certainty)  and  a high  entropy  value  as  a large  uncertainty. 

However,  the  entropy  used  in  attribute  selection  for  building  decision  trees  has  shown  a 
tendency  to  select  many  valued  attributes.  Besides,  the  entropy  value  is  different  when  more 

entropy  for  three  classes  lh-  While  0 < //;  < 1,  the  entropy  H3  satisfies  0 < Hy<  log( 3). 
Most  of  those  problems  were  documented  by  Quinlan  and  Argucllo  [31],  [4]. 

The  information  gain  criterion  or  entropy  (equation  3.1)  can  be  used  to  this  aim,  and 
its  certainty  is  given  by: 


C(p)  = 


(5.2) 


Since  the  entropy  based  criterion  has  several  limitations  as  shown  by  Quinlan  [31],  I am 


wh'-tttE?; 


(5.3) 


where  pj  = max, p,.  Equivalently,  the  previous  cquatia 


Intuitively,  the  determination  guesses  the  most  probable  class  in  a given  data  set  based 


Figure  5.1.  The  determination  measure 

This  criterion  measures  the  relative  importance  of  the  dominant  class  in  a data  set 
(the  class  with  a higher  relative  probability)  with  respect  to  the  remaining  classes.  If  the 


it  are  lower-,  then  the  determination  is  lower.  On  the  contrary,  if  the  probability  of  the 


Figure  5.2.  The  determination 


be  higher.  Thus,  a full  dominant  clans  leads  to  a determination  of  1 and  the  absence  of  a 

From  a measure  point  of  view,  we  are  measuring  the  difference  of  dominant  class  with 

values.  Since  the  value  can  be  as  large  as  the  dominant  class  value;  then  we  normalized 
dividing  by  zero.  Thi6  measure  is  therefore  similar  to  the  square  error  measure  taken  over 
the  n - 1 non-dominant  components  and  then  normalized.  A potential  different  measure 
can  be  obtained  using  the  square  error  and  normalizing.  We  prefer  the  simpler  and  easier 


The  simplicity  of  the  determination  formula  allows  an  easy  interpretation  of  its  respec- 
tive values.  For  example,  although  an  80%  of  certainty-based  entropy  indicates  almost 
nothing  about  the  nature  of  the  dominant  class,  an  80%  determination  indicates  that  the 
dominant  class  is  5 times  higher  than  the  average  of  the  remaining  classes.  Application  of 
the  proposed  formula  shows  that  if  the  determination  is  1 - a,  then  the  dominant  class, 
say  j,  satisfies:  pj  = fj,  a > 0 

or  equivalently: 


fi  = V(pi,pi,--,pn)  = ,UH*  therefore  pmar  = Thus,  when  two  classes 

are  present,  a level  of  0.80  % confidence  can  be  achieved  with  a determination  of  0.75. 
Given  the  distribution  of  probabilities,  one  can  decide  which  is  the  best  determined 

Given  two  sets  with  the  same  relative  frequencies,  a way  to  distinguish  between  them 
is  to  consider  their  sire.  Thus,  the  support  of  a given  data  set  is  the  data  set  size.  The 


This  section  shows  that  it  is  possible  to  derive  mathematically  the  determination  mea- 
sure on  the  basis  of  a limited  sot  of  assumptions,  using  a method  similar  to  Shannon's  for 


Given  a set  of  measures  Pi  > 0,i  = l,..,n,  n > i,  such  that  > 0 i.e.,  one  of  the  Pi 
must  be  nonzero  and  there  must  be  at  least  two  possible  events  if  we  want  to  distinguish 
between  them. 

1.  0SJ>(pi,p> A.)<  1 

2.  0(pi,Ps,— ,pn)  = 0 , if  pi  = p for  ail  i. 

3.  0(pi,Pa Ih.)  = 1.  if  Pi  >0  for  some  i and  p,  = O./orjji  i. 

4.  D(p-Qitp-aa p-o,,_i,p)  = 1 - D{a\,aj II..- 1 , p),  0 < a,-  < p,p>  0 

5.  D(p,,...,pb~,p„~,p»)=  0(pi,..,p„..,pt,..p„) 

6.  D(C  tpi,Ctpi,...,Ctpn)=  D(pi,w,..,P,),C  > 0 

Assumption  1 says  that  the  measure  must  be  be  in  the  range  [0,1]  meaning  that  zero  is 
the  minimum  determination  and  1 is  the  maximum  determination. 

Assumption  3 says  that  in  total  discrimination  -only  one  pi  not  null-  the  determination 
must  be  maximal. 

Assumption  4 forces  an  equal  treatment  for  all  measures  independent  of  their  coordi- 
nates or  indexes.  It  allows  that  under  similar  conditions,  the  change  in  determination  must 
be  the  same.  Note  that  the  vector  (p  — Oi.p  - O], o„_i, p)  is  a distance  d = or 
from  the  vector  (p,p p).  Similarly,  the  vector  (0,0, ...O.p)  is  a distance  of  d of  vector 


corresponds  to  our  intuition  that  D(0.98, 1)  and  D(0.02,l)  are  related  by  D(0.98,l)  - 


D(0.02,l). 

Assumption  5 says  that  the  determination  function  is  completely  symmetric  i.e.,  the 
interchanges  of  any  two  coordinates  should  not  affect  the  result. 

Assumption  6 says  that  multiplying  the  measures  for  any  constant  should  not  affect  the 

Note  that  the  particular  case,  when  £Pi  = 1 represents  a distribution  of  probabilities 
and  therefore  the  determination  can  be  applied  in  the  same  way.  Sometimes  it  is  useful 
to  think  in  normalised  determination  i.e..  when  its  arguments  can  be  seen  as  a set  of 


simple  to  compute  i.e., 


o introduce  a function  that  satisfies  nssuntplit 


Derivation  of  the  Du  n I mill 


Theorem  1: 

Assume  that  the  Determination  formula  is  of  the  following  form:  Dll, , xs)  = Q;x I r'  -I 


Using  condition  3 and  6;  D(k, 0)  = £,•«;  •*'•  + /!=  1 


D(0,k)=£ibi,k‘-  + B = t 
are  equal  if  ail  their  coefficients 


Then,  by  condition  2,  D(Ct  C)=  1 + £<  ' = 0 and  again  for  condition  G -this 

Since  the  exponents  are  all  positive,  such  a condition  is  not  possible,  and  hence,  there  is  no 
such  polynomial. 

Theorem  2: 

D(Xi,Xi)  = max(l  - xi/xj,  1 - xj/i|) 

(taking  division  by  zero  as  a limit  to  positive  infinity). 

Without  loss  of  generality,  let  ns  assume  that  0 < xi  < xj.  Thus,  ) = 1 — 

since  1 - xj/xi  < 0 (or  its  limit  when  xi  tends  to  zero.) 

Thus,  assumption  1 holds  : 0 < 1 — < 1 

Assumption  3:  0(0.  xj)  ss  1 — 0/x2  = 1 for  every  x?  > 0. 

Assumption  4:  H(zj  — o,  xj)  = 1 - = 1 - (1  — ^)  = 1 — f?(t»,  xj) 

Assumption  5:  The  interchange  of  variables  doesn't  change  the  sign  of  the  inequality 


ic  uniqueness  of  the  function,  but  simply 


n > 0,  no  polynomial  mi 


A possible  polynomial  function  can  be  expressed  in  the  following  form: 


m = DE'-mp?-') + E MIK1*) + * (s-5) 

dj  / 0,Vi,j  (5.8) 

and  there  are  at  least  two  Sf.r  > 0 for  a given  k. 

D(C.e-i)  = E.OUC,-  + /f  = l 

This  must  be  true  for  every  C > 0,  then  for  5.6  all 


(5.7) 


Then,  by  condition  2, 


and  again  for  condition  6 -this  is  valid  for  all  constant  vectors  C-,  there  should  exist  a 


(5.10) 


Since  there  exist  at  least  twos;,*  > 0 for  each  k,  5. 10  can  not  hold  and  such  a polynomial 
Theorem  4: 

Given  n > 0,  a measure  that  satisfy  conditions  I through  6 is  D(p)  = 1 - 2c, 

where  p,  = maxip,1 

Then. 


and  thus,  D(p)  > 0. 

D(p)  < I. 

Assumption  2: 

O(P)  = 1 - T=ilT<"  - 
Assumption  3: 

0(0,0 o.p.)  = 1 - j^ij  . 0/p„  = I 

0(P  - o„ P- on, P-  a P)  = 


positive,  which  implies 


e p,  = P for  aU  i. 
P/P=  0 


<‘-(^■[£.*,70  = 


1 - D(a„a, o„.„n 

The  interchange  of  variables  doesn't  change  the  sign  of  the  inequality  c;  < xn,  > 
assumption  5 holds. 

D{C  • p)  - 1 — for  C > 0,  Cpn  is  still  maximum. 

Therefore.  D(C  • p)  = /J(;7|. 


In  this  section,  I show  how  the  determination  can  be  use  to  rank  rules  for  rule  induction. 
The  genera]  problem  is  defined  by  the  following: 

Smyth  and  Goodman  used  cross-entropy  to  evaluate  rules.  Here  is  a comparison  of  the 
determination  measure  and  its  use  for  ranking  rules  with  the  measure  supphed  by  Smyth 


j(Xi  y = S)  = pWsl/osll^fi)  + ( 1 - p{x/y))log( and  the  J-measure: 

J(X\  Y = p)  = p(s)j( X;  Y = If)  [41]. 

dcl(X ; Y = s)  = inax(  I - 1 '’/$)  and  Det{X;Y  = y|  = p(y)d't(X ; Y = y) 

The  following  example  is  due  to  Smyth  and  Goodman  [41,  pp  164,165]  (I  have  added 
determination  measures  to  the  tables  ). 


Table  5.1.  Joint  probability  distribution  for  Medical  Diagnosis  example 


Table  5.1  shows  the  probability  distribution  of  medical  cases  for  diagnosis  of  a Disease 
X.  Table  5.2  shows  a set  of  potential  rules  and  the  evaluation  of  each  rule  using  both: 
the  J-measure  and  the  determinal  o n l res  shown  above.  The  similarity  between  both 

n.  This  is  the  case 


It  is  worth  noting  that  the  previous  set  of  rules  (except  rules  2 and  6}  can  be  seen 
precedent  and  the  leaves  represent  the  Una]  outcome. 


S.3  Application  to  Classification  in  Large  Databases 
•M  l Influence  of  Many-Valued  and  Irrelevant  Attribute? 

Despite  many  studies  that  show  the  tendency  of  the  entropy  to  favor  many-valued 

The  other  three  experiments  use  a modified  version  of  the  entropy  [31)  called  the  gain-ratio 


and  similarly  a modified  version  of  Determination  to  favor  few-valued  attributes.  It  will  be 


The  data 

The  synthetic  databases  for  I he  experiment  consisted  of  20000  cases  each,  20  attributes. 
There  were  -1  databases,  the  two  first  databases  for  the  first  part  of  the  experiment.  The 
last  two  database,  together  with  the  first  one  are  used  for  the  second  part  of  the  experiment. 


1.  The  data  generation  program  was  instructed  to  generated  10  class  values  and  to  leave 


(A15  to  A 19)  have  a range  of  30  for  attribute  A IS  and  250  for  attributes  Aid  to  A19. 
The  remaining  attributes  were  relevant  to  the  class  value  and  all  of  them  have  2 or 
3 values.  The  data  were  generated  using  a modified  version  of  the  DGP/v2  of  P. 
Benedict  [7],  Despite  the  random  nature  of  the  program,  there  is  no  guarantee  that 
a random  dependency  of  the  Class  attribute  with  the  irrelevant  variables  can  not  be 


2.  In  this  database,  10  attributes  were  left  as  irrelevant  and  to  facilitate  the  induction 

500  values.  While  in  the  previous  database,  the  five  many-valued  attributes  had  little 
chance  to  be  chosen;  in  this  database  the  10  many-valued  attributes  had  a major 


3.  Again,  10  classes  were  generated  and  this  time  there  were  10  irrelevant  attributes. 
Prom  the  10  relevant  attributes,  five  were  chosen  as  many-  valued  (A5  to  A9  with  250 


values)  and  from  the  10  irrelevant  attributes  five  we 
to  A19  with  200  values). 


iterations,  starting  with  a sample  sire  of  10%  (2000)  of  the  cases.  The  error  rate  and  soft 
error  rate  (sometimes  called  the  error  caused  by  undefined  cases  i.e.,  error  caused  for  cases 


Table  5.3. ' 


the  sample  for  entropy  while  determination  tends  to  get  a lower  error  rate  after  the  sceond 

A1S.  After  that,  it  must  he  noted  that  while  the  determination  stichs  to  the  same  attribute 
(A15  with  30  values),  entropy  favors  A18  with  250  values.  At  the  end,  entropy  changes 

corresponds  to  the  root  of  the  final  tree  for  determination.  It  is  interesting  to  note  that 
even  though  A15  was  marked  as  irrelevant,  the  final  fact  is  that  there  is  an  association 
between  A15  and  the  dass  (as  the  decision  tree  says).  I believe  this  was  primarily  due  to 
the  few  values  of  AI5  (30)  and  to  a coincidence  of  the  normal  distribution  used  for  the  data 


generation  program.  Note  that  Ala  is  still  important  as  a closest  attribute  for  the  root  in 
Table  5.3.1  for  entropy. 

In  a second  trial,  the  second  synthetic  database  was  used.  Figure  5.5  shows  the  results 
for  12  or  13  iterations.  However,  the  error  rate  is  lower  in  both  cases,  due  to  the  lower 
number  of  classes  (2);  the  determination  error  rate  is  generally  the  lowest.  Table  5.3.1 


is.  While  both  criteria  choose  as  roots  a few-valued 
nodes  that  some  many-valued  attributes  were  chosen  as  subroots  in  the  subtrees  and  hence 


The  second  experiment 

ria  [31].  Thus,  for  the  selection  step  of  the  tree  construction 


suggested  the  gain-ratio 


algorithm,  the  best  attribute  is  selected  as  the  attribute  A that  minimizes: 


(■»„($) -£fl„(A)) 

rv{A) 


(5.11) 


//„(£)  Is  the  entropy  according  to  the  class  distribution  of  the  data  set  of  instances  S. 
EHn{A)  - Y,  P(A  = a r)  * ff„(A  - u.)  (average  class  entropy  for  the  attribute  A ). 
!V{A)  is  the  randomness  measure  of  the  partition  caused  by  A i.e.,  the  entropy  of  the 

Note  that  this  equation  will  favor  low-valued  attributes,  since  the  largest  value  of  IV(A) 


A similar  component  is  introduced  here  for  the  determination  formula,  the  root  attribute 


D,(P)  = 


m 


. m 


(5.12) 


\A\/MC 

where  MC  = maxs  |A|. 

Note  that  this  equation  reduces  the  determination  of  au  attribute  according  to  its  rel- 


Using  the  first  database,  several  iterations  were  made  for  both  criteria.  Figure  5.6  shows 
the  error  rate  for  both  few-valued  determination  and  the  gain-ratio.  In  this  case,  the  gain- 


ratio  tends  to  outperform  the  determination  modified,  but  the  soft  error  rate  is  still  lower 
for  determination.  Note  that  the  reduction  in  the  error  rate  from  the  figure  5.4. 


Table  5.3.1  shows  the  roots  selected  in  the  first  six  iterations.  The  criteria  effectively 


5.7  shows  the  error  rate  for  both  few-valued  determination  and  the  gain-ratio.  In  this  case, 
the  gain-ratio  outperforms  the  determination  mollified,  hut  the  soft  error  rate  is  still  lower 
for  determination.  The  behavior  of  the  gain-ratio  is  consistent  but  determination  behaves 

Thhle  5.3.1  shows  the  roots  selected  in  the  first  six  iterations.  The  gain-ratio  criterion 


effectively  chooses  few-valucs  attributes  rather  than  many-valued  attribute  as  expected. 


Figure  5.7. 


Note  that  A 13 is  a few-valued  irrelevant  attribute  (thought  it  helps  in  the  final  classification 


were  chosen  as  subroots  in  the  determination  experiment  and  hence  the  large  error  rate. 


Using  the  third  database,  several  iterations  were  made  for  both  criteria.  Figure  5.8 

Note  the  tendency  of  the  both  errors  of  the  entropy  to  be  "inside"  between  both  errors 
of  determination  for  the  Figure  5.5,  5.7  and  5.8. 


Figure  5.S.  Many  values  Experiment 


Table  5.3.1  shows  the  roots  selected  in  the  first  six  iterations.  The  criteria  effectively 

consisted  fo  few-valued  irrelevant  attributes  rather  than  the  relevant  attributes  (hence  the 
large  error  rate). 

In  conclusion,  when  there  are  irrelevant  many-valued  attributes  present: 


2.  Entropy  tends  to  be  erratic  with  small  samples  relative  to  cardinality  of  the  many- 
valued  attribute  domain. 


rate  is  still  high  because  of  the  influence  of  many-valued  attributes.  However,  this 


‘I.  The  missdassi  Real  ion  error  is  given  Tor  the  difference  between  the  two  curves  in  each 
criterion.  Entropy  seems  to  have  a low  missdassification  error  but  a large  soft  error 
(undefined  error)  in  all  cases. 

5.  Many-valued  attributes  tend  to  be  chosen  as  roots  mainly  in  subtrees  when  the  size 

0.  Although  the  experiment  was  conducted  with  a relatively  small  set  of  20000  cases, 

many-valued  attributes.  Eventually  any  large  data  base  will  be  partitioned  in  small 
subsets  and  the  induction  on  those  will  he  affected  for  many-valued  attributes.  Even 
though,  the  error  rate  in  large  databases  will  be  only  lightly  affected  because  relevant 


The  next  experiment  was  designed  with  the  aim  to  compare  the  inductive  ability  of 
simple  determination  with  the  entropy  in  an  environment  where  the  entropy  (and  determi- 


erated  for  the  experiments.  Each  database  consisted  of  20  attributes  (A0  to  A19),  the 
class  attribute  and  10  values  per  attribute  (0  to  9)  approximately  -avoiding  the  effects  of 


1 assigned  determines  the  type  of  the  database  as  described  below. 


The  first  database  includes  just  two  classes.  The  main  class  consists  of  all  those  instances 
around  a random  peak  in  the  20  dimensional  space  generated  by  the  data  generation  pro- 
gram developed  by  Powell  Benedict  and  others  (7]. 

The  second  database  was  developed  using  the  same  program  but  modified  to  generate  10 
class  groups,  to  substantially  complicate  the  task  of  the  decision  tree  induction  algorithm. 

The  classes  in  the  third  database  were  generated  at  random.  Actually,  one  attribute 

For  the  last  database,  classes  were  designated  using  a decision  tree  known  beforehand. 
An  initial  database  was  generated  and  class  values  were  changed  accordingly  to  the  decision 
tree  output.  This  case  represents  a database  that  has  a well-defined  and  known  decision 


5.4.2  Experiments 

Four  experiments  were  conducted  to  demonstrate: 


the  entropy-based 


• The  applicability  of  decision  tree  approach  to  large  databases  (as  they  have  been 


The  effectiveness  of  the  use  of  a small  sample  set  (instead  of  the  entire  database)  for 
knowledge  discovery. 


s.  Each  tree  induction  was  determined  for  the 


; part  of  the  table  and 


to  17  %)  taken  from  the  database.  Once  the  initial  decision  tree  is  derived  for  the  sample 
set,  the  rest  of  the  database  is  tested  against  the  tree  and  the  error  rate  computed.  Then 
a percentage  of  the  exceptions  is  used  to  reconstruct  the  decision  tree  and  again  the  rest 

of  iterations  (one  in  some  cases).  The  process  was  halted  either  if  the  error  rate  was  low 
enough  or  if  only  a slight  improvement  over  the  previous  error  rate  was  computed  for  the 


scribed  in  terms  of  the  induced  decision  tree.  On  the  other  hand,  reasonable  improvements 
the  appropriate  decision  tree  for  the  entire  data.  The  above  experiment  is  designed  to 


3-1,3  Updating  the  Window  through  Sampling  of  Exceptions 

to  incorporate  a small  percentage  of  the  exceptions  in  each  iteration.  A small  sample  that 

talion,  a parameter  to  the  induction  process  is  provided  to  select  this  small  sample  of  the 
exceptional  cases.  The  reason  behind  this  is  to  keep  the  window  size  small,  since  many 
exceptions  can  he  due  to  the  same  cause  (a  wrongly  labeled  leaf  or  a missing  branch).  This 
can  lead  to  a slow  convergence  in  some  cases  but  it  avoids  superfluous  data  in  the  window. 


t:  The  initial  .sample  set  with  which  the  induction  process  starts.  Ail  sample  < 


taken  uniformly  distributed  over  the  synthetic  database.  This  guarantees  a meaning- 
ful sample  from  the  database.  The  table  indicates  t hose  cases  where  a different  initial 
sampling  method  was  used. 


Initial  error:  The  initial  classification  error  when  the  rest  of  the  database  was  tested 
against  the  tree  derived  for  the  initial  sample. 

N.  |t.:  Number  of  Iterations  done  to  get  a final  tree  (by  including  the  exceptions  added 


% Ex.  Sd.:  % of  Exceptions  Sampled:  Percentage  of  exceptions  that  are  added  to  the 


Final  error:  The  final  tree  error  measured  with  the  rest  of  the  database. 


Final  S.  Size:  Final  sample  size  that  includes  all  the  exceptions  that  were  added  in  each 


Tree  Size,  Tree  IIt.(  Height).  Tree  Leaves,  Tree  Nodes  : 


Root  and  Root  Meas.:  The  decision  tree  root  attribute  and  the  measure  - either  deter- 


Ext.  TV.:  The  number  of  external  trees  required  when  the  tree  no  longer  fits  in  memory. 
Only  the  main  body  of  the  tree  is  left  in  memory. 


Other  terminology  used  but  not  shown  in  the  table: 


Soft  cases:  tree  exceptions  due  to  missing  tree  brandies.  These  constitute  a main  source 
of  exceptions  when  dealing  with  attributes  that  have  large  domains. 


Experiment  #1.  Database:  100000  records,  two  classes,  20  attributes,  10  values  per 
attribute.  See  tables  5.8  and  5.9. 

Table  5 8.  Exp.  1.  Criterion:  Determination 


In  all  cases  soft  errors  make  up  40%  of  the  final  error. 

rows  was  used  to  test  the  trees.  First,  1 145  ( 1357  for  entropy)  rows  were  added,  and 
then  787  (445  for  entropy)  rows  were  added.  The  approach  was  abandoned  since  1 
was  approximating  the  tree  from  the  5000  rows  sample  which  had  a 0.19  and  0.15 

(2)  13  subtrees  having  an  average  4 nodes,  4 leaves,  and  1 level  wore  stored  in  external 
files  (secondary  storage).  The  average  tree  shre  was  4k  bytes. 

(3)  19  subtrees  having  an  average  of  4 nodes,  3 leaves,  and  1 level  were  stored  in  external 
files.  The  average  tree  size  was  3k  bytes. 


’)  A posterior  check  of  the  decision  tree  shows  that  A16  has  the  sa: 
that  of  A3  (A3  was  chosen  by  lexicographical  order). 


- Final  errors  can  be  reduced  by  almost  40%  if  the  missing  branches  corresponding  to  the 

- To  obtain  11%  (say  from  23%  to  14%)  improvement  in  error,  it  Is  necessary  to  increase 

the  sample  size  by  almost  7 times  (from  2000  to  14000). 

behavior  while  there  is  more  stability  of  the  tree  for  determination.  The  final  trees 

Experiment  #2:  Database:  100000  records,  10  classes,  20  attributes,  10  values  per 
attribute.  See  tables  5.10  and  5.11. 


errors  make  up  50%  of  the  final  i 


(1)  Subtrees  having  an  average  3 or  4 nodes,  3 leaves,  and  1 level  were  stored  in  external 

(2)  Subtrees  having  an  average  4 or  5 nodes,  3 leaves,  and  I level  were  stored  in  external 
files.  The  average  tree  size  was  4k  bytes  with  slight  variations. 

• Again,  certainty  based  entropy  and  determination  tends  to  lead  to  different  decision  trees 
but  with  similar  structures  (nodes,  leaves,  height  and  sire). 

mination  but  both  the  trees  are  of  similar  accuracy  (28%  to  30%). 

- Larger  samples  were  not  analysed  since  they  require  many  iterations  for  a significant 

Experiment  #3:  Database:  100000  records,  10  classes,  20  attributes,  10  values  per 
attribute.  A random  class  definition.  See  tables  5.12  and  5.13. 

In  all  cases  soft  errors  make  up  40%  of  the  final  error. 

{ 1 ) Subtrees  having  an  average  3 nodes,  3 leaves,  and  1 level  were  stored  in  external  files. 


The  average  tree  size  was  3k  bytes. 


(2)  Subtrees  having  an  average  5 nodes  (several  trees  with  7 or  20  nodes),  3 leaves,  an 
level  were  stored  in  external  fdes.  The  average  tree  size  was  4k  bytes. 

- Both  criteria  behave  similarly.  Their  ability  to  correctly  classify  the  cases  are  equ. 
bad  (due  to  the  random  class  assignment). 


In  all  cases  soft  errors  make  up  100%  of  the  final  error. 

(1)  This  set  constitutes  the  first  172  cases  of  the  artificial  database. 

. Definitely,  the  embedded  decision  tree  was  detected  easily  for  both  criteria.  A very  small 
sample  of  172  (0.2%)  leads  to  an  almost  exact  decision  tree  (0.018%  error). 

- Even  a bad  sample  (set  10)  leads  to  an  exact  decision  tree  after  2 or  3 iterations  in  both 


Figure  5.9  shows  the  relationship  between  sample  size  and  error  rate  for  both  determi- 
ne., there  are  not  many  valued  attributes  present  which  interfere  with  the  induction  process. 

a good  alternative  to  extract  rules  without  the  expense  of  processing  the  whole  database. 
However,  this  can  be  done  with  any  attribute  selection  criteria;  it  is  important  the  com- 
puting time  invested,  the  behavior  of  the  attribute  selection  criteria  in  the  presence  of 
many-valued  attributes  and  its  relationship  with  confidence  and  support  . Determination 
is  fast  to  compute  than  entropy  and  can  be  easily  adapted  in  the  presence  of  many-vtdued 


attributes  ( /?(p)  and  0/(p)  coincide  if  all  attributes  has  the  same  cardinality).  Besides,  the 
undefined  error  (soft  error)  when  many-valued  attributes  wore  considered  was  smaller  than 
the  entropy  case.  This  suggests  that  the  determination  will  fit  better  the  domain  of  cases 
however  its  ability  to  classify  correctly  cases  inside  the  decision  tree  domain  is  sometimes 
much  lower  than  entropy  (classification  error)  leading  to  a larger  error  rale  in  some  exper- 
iments. The  fact  we  can  relate  the  node  determination  measure  with  the  confidence  and 
support,  allows  for  an  easy  interpretation  of  observed  results  (or  classification)  as  compared 

to  an  inexact  (50%  error  or  more)  decision  tree. 

The  experiments  were  carried  out  on  a Sun  Workstation  with  8 megabyte  of  memory  in 
a multi-user  environment.  Execution  times  were  around  20  to  30  minutes  for  deriving  the 

Pentium  based  PC  computer. 

gest  that  determination  is  a viable  alternate  criteria.  The  biased  criteria  (gain-ratio  and 
few- value  determination)  can  selected  few-valued  attributes  that  are  not  important  or  rel- 


CHAPTER  6 

DECISION  TREES  AND  ASSOCIATION  ROLES 


Decision  Trees.  Functional  Dependent 


Knowledge  Discovery  consists  mainly  of  finding  rules  among  data.  Here  I formalize 
the  concept  of  association  rules,  their  relationship  to 


fi.1.1  Confidence  and  Support  in  Decision  Tree? 

Definition  / A path  P in  a daemon  tree  is  a sequence  of  attribute-value  paint  denoted 


Definition  3 The  cottfidenee  on  the  decision  represented  by  a leaf  L{P)  is  denoted  byC(L(P)) 
denoted  byV(L{P))  . 


Definition  J The  support  of  the  decision  represented  by  the  leaf  L(P)  is  given  by  its  cardi- 


fi.1.2  Definition  of  Association  Rules 


Date  describes  a functional  dependency  among  attributes  or  features  in  a database  as 
an  attribute  B depends  functionally  on  an  attribute  A,  if,  for  every  value  of  A,  every  tuple 
that  contains  this  value  of  A,  always  contains  the  same  value  for  B [9]. 

base  and  t.B  denotes  the  value  of  attribute  B in  tuple  r: 

Vu  € P(A),Vr,/i  e U such  that  a € r,a€  p=o  T.B  = p.B 
This  functional  dependency  (fd)  is  denoted  as  A ~ II 

view  of  Knowledge  Discovery. 

First,  the  data  base  U is  generally  dynamic.  We  don't  know  all  tuples  in  a given 

most  of  them.  This  is  not  a problem  since  we  can  consider  a more  restricted  domain  for 

like  to  discard  those  values.  Again,  the  mathematical  definition  does  not  hold,  but  the 

Let  the  “large  known  set  of  tuples"  S be  the  support  set;  and  “the  large  subset  of  the 
known  tuples"  C.  the  confidence  set.  Thus,  the  concept  of  an  association  rule  can  be 


Table  6.1.  Medical  Diagnosis  example 


Let  N the  set  of  natural  numbers.  Given  values  c € [0, 1]  and  s € M, 
If3SCtf/|S|>sand3CcS/|C|>c.|S| 
such  that 

if  H = (5  € Z>(4)/3r  € S,  rj 

( the  set  of  values  of  A restricted  to  the  sot  of  tuples  S),  then 

Vo  € 72  Vr,p€  C/  a € r,o  € p t.B  = p.B  and  Vr.p  € *S\C/o€r,o€p=S  T.B  ^ p.B 
confidence  c in  U. 

The  notation  A •—  B (s,  c)  will  be  used  to  denote  this. 


The  rule: 


has  support  ^ and  confidence  0.75.  The  support  set  is  (7,8,9, 10)  and  the  confidence  set  is 
(8,9,10). 

The  rule: 

| “Symptom  A = fever  A Symptom  B=  sore  throat  ~ Disease  X=presenF~| 

The  rule : 

| "Symptom  B - no  sore  throat  *—  Disease  X=present"  | 

Theorem  I A — B if  and  only  if  /! ' — . II  { | V(U)  |.  1.0  ). 

6.1.3  Association  Rules  in  Pecisjon  Trees 

and  decision  trees.  Those  theorems  establish  this  relationship: 

Notation:  Let  7>,(  A)  a decision  tree  which  classifies  A i.e.,  A is  the  target  (goal)  attribute 

A — Baa  3D,  (fl)/  V,(B).  root  = A A V,{B).hcighl  = 1 

Theorem  .7  Let  D a compound  determinant  attribute.  D = (A],  A3, ...,  A„) 

»B8A  V,  (B)/  'dj,D,(B).aubtree(i,j).root  = A,  A V,{B).l,eight  = n 

Theorem  i The  height  of  the  smallest  decision  tree  is  less  or  equal  to  the  number  of  at- 


Theorem  2 guarantees  that  there  is  a tree  if  there  is  a simple  functional  dependency  of 
the  goal  attribute. 


Figure  6.1.  Illustration  Theorem  2 


A 


Figure  0.2.  Illustration  Theorem  3 


Theorem  4 is  just  a corollary  of  the  previous  theorems  and  limits  the  height  of  the 


Theorem  2 can  be  extended 


Theorem  5 Let  A a simple  attribute . c € (0,  l|,a  € At 
A — B(s,c ) » 3 V,(B)I  Vii  ft). root  = A A V,{B).beighl  = 1 and  372  C D(-4)/ 


e<£C<i({A  = a)))P(A  = a)  (6.1) 

»<S>«>t=-})l  (6-2) 

Proof:  A I — B(s,c)  ey.  From  Theorem  2,  Vi(B).rool  = A A V,{B).h right  = 1 Let  <S,C 
and  R as  in  the  definition  of  the  association  rule.  A and  B coincide  in  every  tuple  in  C . 
Let  c„  the  number  of  tuples  of  C for  value  a of  /f.  and  let  the  number  of  tuples  of  S. 
Then,  | L({A  = a))  |=  «.  and  C(L({A  = u}))  = J;  P(A  = a)  = ft 


^C(L((A  = u}))P(A  = u)=5;i^T  = ||l>c  (6.3) 

£|L(f,4  = o))|=£..=|S|>.  (6.4) 

A~B(s,c)/<»3flA2>,(fi)/ 

Vj,V,{B).subtree(i,j).root  = A,  A V,(B).hcight  = n 


nrfafccJH-W 


c<£c<«{/1  = ..)))P(a  = <>) 


(6.S) 


£!*«-*  = •)>  I (6-«) 


Proof:  The  attribute  A can  be  considered  as  a single  attribute  with  value  a = 

{oi.fla,  ...,«„}  in  every  tuple.  By  previous  theorem,  equations  6.1  and  6.2  coincide  with 
equations  6.5  and  6.6.  The  decision  tree  has  height  one  and  as  root  the  composite  at- 

6,jL-UAudUngJdMyi.\MMiLAtUiimlss 


valued  attributes.  The  intuition  behind  this  is  that  for  small  sets,  an  attribute  with  many 
values  behaves  almost  as  a primary  key  attribute  and  hence  its  determination  of  the  class 
attribute  is  lOO/r.  Many-valued  attributes  aflect  the  resulting  set  of  derived  rules  since 
they  are  not  relevant  to  the  class  determination,  like  the  patient  identification  in  a data 


set  of  diseases.  Continuous  attributes  are  special  cases  of  many-valued  attributes  (every 
continuous  attribute  is  always  represented  by  a very  long  sequence  of  discrete  values).  A 
concern  exists  in  developing  techniques  for  attribute  selection  that  are  not  greatly  influenced 
by  the  attribute  cardinality  such  as  the  1DL  system  of  Van  de  Velde  [10)  or  techniques  that 


can  split  up  the  attribute  range  to  minimize  the  number  of  brandies  in  the  dedsion  tree 
such  as  the  gini  index  [8]  or  the  Kolmogorov-Smirnolf  distance  by  two  classes  [13],  recently 


The  determination  measure  is  not  completely  free  from  being  affected  by  many-valued 
attributes  . My  concern  has  therefore  been  to  incorporate  in  dedsion  trees  a way  to  de- 
crease the  range  of  the  many-valued  attribute  but  at  the  same  time  increase  or  keep  its 
determination  - whidi  is  not  always  possible.  This  means  that  each  branch  of  the  decision 
tree  will  be  labeled  with  a range  (even  unitary  ones)  that  represent  the  set  of  values.  Note 
that  attributes  are  not  recoded,  but  that  a range  is  just  used  as  a label  of  the  respective 

This  range  compression  tedmique  -grouping  together  values  of  the  attribute  that  rnax- 


gel  a more  compact  tree  and  a set  of  derived  rules. 


6.2,1  Ilic  Best  Split  Parliljpa  Algorithm 

Let  M{j.  k)  be  any  positive  measure  over  the  values  j to  k. 

Let  II.  - {ji,  jj, a set  of  partition  points  over  the  set  of  values  !’] 


if  p < 9 and  Ji  = 1 . . 


Let  p(s,  k)  be  the  probability  of  range  e.  to  e*: 

<6-7> 

Let  M(nr)  be  the  average  measure  over  IIr: 

Af(II,)=  53  pOii- ufxW (Jli-l . M (6-8) 

Definition  5 II,  is  an  optimum  partition  if  it  maiimiees  the  value  of  X/(IIr)  with  a mini- 
mum number  of  intervals  I.C.,  if  there  is  another  partition  with  the  same  value  of  M then 


M(nr)  = rtnl)tf(n!)+p<n?)M(n?)  (6.9) 


Best  Split(integer  I,  integer  N): 
sO  Max=  M([v(I),v(N)J);  //the  complete  range 


1 = Besl  Split(I,P); 

! = Besl  Split(P+l,N); 

= p(I,P)*bl  + p(p+l,N)*b2; 


If  MiMax  llien 
Max  = M; 


Correctness  of  the  Besl  Split  Partition 

Theorem  S The  Bent  Split  algorithm  finds  the  optimum  partition. 


Basic  case:  m=l.  If  nf  is  not  optimum,  assume  that  Ilr  is  optimum  ( r > 1);  then 
M (II,)  > Af([ui,  e„))  = Af  (nf );  but  steps  si  and  sl.t,  together  with  theorem  7,  guarantee 

Induction  Hypothesis:  nf  is  optimum  for  i < m 


le  that  II,  is  an  optimun  partition.  First,  Ilf  can 
be  seen  as  the  concatenation  of  the  first  two  partitions  {ii,„ ip}  and  {jp+i, where 

Assume  n,  = {or , 02,  Or)  then,  we  have  two  cases: 

theorem  7 [01,02]  and  [03,  ...o,]  are  optimum  subpartitions  and  with  maximum  value. 


- 02  > jpi  then  by  a similar  argument  Best  Split  must  have  detected  02  after  finding  jp. 


Assume  we  have  two  classes.  The  following  table  is  the  attribute  value  distribution  for  each 


Analyzing  first  partition  of  1,2,3, 4: 

[1]  [2,3,4]:  1:0.5  (3) 

2,3,4:  2:  1.0(1) 

Analyzing  optimum  for  3,4: 

3,4:  3:  1.0(2) 

4 : 0.0  (2) 

[3], [4] : l/4(  1.0*2+  0.0*2)  = 0.5 
[3,4] : (1-1/3)=  0.60  (4)  (a) 

Optimum  for  3,4  is  [3,4]  with  value  (1.66. 
Evaluating  first  partition  of  2,3,4: 

[2], [3,4]  = 1/5  ( 1 + 4 * 0.66)  = 0.73  (max) 

Analyzing  optimum  for  2.3: 

2,3  : 2 : 1.0  (1) 


3:  1.0(2) 


[2],[3] : l/3(  1*1.0+  2*1.0)  = 1.0  (3)  (b) 
[2,3] : (1-1/2)=  0.5  (3) 

Optimum  for  2,3:  [2],  [3] 


4 : 0.0  (2) 

[21, [3], [4]  = 1/5  ( 3*1.0  + 2 * 0.0)  = 0,60 

Then,  the  optimum  for  2,3,4  is:  [2], [3, 4]  with  value 
l/8(  3 *0.5  + 5 * 0.73)  = 0.644 

Analyzing  second  partition  of  [1,2*3, 4]: 

[1,2]  [3,4] : 1,2  : 1 : 0.5  (3) 

2:  1.0(1) 

[1],[2] : 1/4  ( 3*0.5+  1*1.0)  = 0.625  (4) 

(1,2]  : 1 - 1/3  = 0.666  (4) 

[3,4] : 0.666  (See  (a)  above) 

: l/8(  4 *0.666  + 4 * 0.666)  = 0.666  (optimum) 

Analyzing  third  partition  of  [1,2, 3.4): 

[1,2,3]  [4]:  1,2,3  : 1:  0.5(3) 

2,3  : [2], [3] : 1.0  (3)  (See  (b)  above) 

[1),[2],[3] : l/6(3*0.5+  3 *1.0)=  0.750  (6) 


•I  : 0,0  (2) 


: l/8(  6*0.75  + 2*0.0)  = 0.562 


Best  option:  [l,2l,[3,4|  with  value  0.666. 


The  Best  Split  algorithm  is  useful  tc 

subintervals  instead  of  the  optimum  partition  of  each  subinterval  to  choose  the  partition 
point  or  split  point.  This  saves  time  in  the  computation  of  range  compression  but  it  does 
not  guarantee  an  optimum  range  compression. 

values  and  class  count  (frequency)  distribution  is  given. 

( /(».'»■ 


of  the  attribute. 

Range  Compression  Algorithm: 
sO  [Traverse  the  value  list  and  join  consecutive  values 

and  t'l+i  are  in  the  same  range.  ] 

Group  values  with  same  class  into  ranges. 

best  two-way  spliting  (with  large  determination) 
for  the  accumulated  class  frequencies  ] 

Get  accumulated  frequencies  F(r,-)  where 
F{'i)  = /(r i ) and  F(r,+,  )=  F(r;)  + /(rl+i) 
s2  [ The  best  partition  point  is  this  that  maximises  the 


measure  must  be  larger  than  the  actual  attribute  measure 


Two  lists  are  needed:  one  for  accumulated  class  counts 
until  certain  value  and  other  to  hold  the  accumulated 
class  frequencies  for  the  remaining  values. 

If  Hr,)  is  the  accumulated  frequency  until  range 

-•plil  = max(P(r,...r,)  . A/(t(r,))  + . Af(fl(r.))) 

When  a partition  point  for  the  range  is  found;  the  remaining 

and  to  right  of  the  partition  point  (if  this  is  possible).] 

Get  the  best  partition  points  (if  any)  for  the  set  of  values  (splits) 
s3  Join  all  the  ranges  accordingly  to  the  Splits  list. 

Counts  (frequency)  for  that  range. 


Assume  we  have  two  classes.  The  following  table  is  the  attribute  value  distribution  for  each 


[1]  [2]  [3]  I1!]  [5,61  PI 


ll.  (12. 0.0  + 7. 1.0  + 7. 0,25+ 10. 0.33+  13.  1.0  + 5. 0.333+  11 .0-17  + 6.1-0)  = 


Step  si:  Finding  splits: 


Therefore  , the  first  split  is  in  range  [5,6];  since  0.610  is  maximum  and  greater  than  the 
original  determination  of  0.490. 

Similarly,  there  are  no  splits  from  1 to  [5,6]  [calculations  not  shown)  while  there  are 
splits  in  [7]  and  [9].  After  joining  all  those  ranges,  the  final  ranges  will  be  [1,6],  [7]  and 


As  an  example  of  what  range  compression  can  do,  an  artificial  database  with  10000  cases 
was  generated  for  two  classes,  with  attributes  in  the  range  of  1 to  1000.  A sample  of  2000 
rows  was  used  to  generated  two  decision  trees,  one  without  and  one  with  range  compression. 


Then  the  decision  trees  were  used  to  extract  the  moximun  support  conjunctive  rule  (MSC 
rule)  given  for  the  tree.  This  is  the  branch  of  the  tree  with  hare  a larger  set  associated  to 

The  terminology  is  the  same  used  in  chapter  5 on  page  69.  In  the  first  case,  the  Initial 


error  was  due  to  67%  soft  cases  since  the  long  range  of  attributes  was  not  considered  in 
the  tree.  In  the  range  compression  case,  the  soft  error  was  reduced  to  35%,  and  hence  the 
reduction  in  the  Initial  error  to  51%.  In  addition,  the  first  decision  tree  leads  to  a MSC 

| If  Al=502  then  class  = 1 (12,  l-OJI 
and  the  second  tree  leads  to  a MSC  rule: 

| If  A1  > 633  then  class  =0  (23, 1.0)  | 


However,  both  rules  are  still  useful,  since  each  one  describes  a different  class. 

In  a separate  test,  a 100000  case  database  for  two  classes  was  generated.  Ear 
tribute  had  128  potential  ralues.  The  trees  we 


The  external  trees  for  both  cases  were  similar  in  shape  to  previous  cases:  4 node  trees 
with  3 leaves  on  average.  The  maximum  support  rule  derived  for  the  first  tree  was: 

| If  AO  is  in  1 102, 102 ) then  class=  0 (21, 1.0)  | 

There  were  0.532  % soft  error  cases  for  the  simple  tree,  for  a final  net  error  of  13  96.  Mean- 
while, there  were  0.342  % soft  error  cases  for  the  tree  with  range  compression,  for  a final 
net  error  of  20  %.  The  maximum  support  rule  derived  for  this  tree  was: 

| If  A0  in  fn  [ 21, 41  ] then  Cl.ass=  0 (122, 1.0)  | 


Both  trees  had  the  sai 


* root  with  the  same  measure,  which  nn 


when  the  data  base  was  generated.  However,  the  tree  with  range  compression  was  small 


CHAPTER  7 

COMPARISON  WITH  OTHER  SYSTEMS 


There  are  several  approaches  to  solve  the  classification  problem  in  Knowledge  Discovery. 
Among  others,  the  ITI  system  from  Utgoff’s  |45]  is  a complete  system  to  induce  trees 
incrementally.  However,  the  trees  are  kept  memory  resident,  and  hence  the  system  is  not 

At  the  other  extreme  is  the  SLIQ  system,  which  derives  decision  trees  for  large  amount 
of  data  and  hence  keeps  most  of  the  data  off  line.  Both  systems  represent  the  latest  in 
tree  construction.  As  SLIQ  is  an  important  approach  to  decision  tree  construction  in  data 
mining  for  very  large  data  bases  I compare  approach  described  here  with  the  SLIQ  approach. 
See  chapter  2 for  a description  of  the  SLIQ  approach. 


I.  SLIQ  requires  at  most  two  passes  over  the  data  base  for  every  level  of  the  decision 


decision  tree  algorithms)  but  it  doesn't  apply  any  incremental  approach.  Actually,  it 


is  not  dear  how  an 


i can  be  integrated  with  their  algorithm. 


many-values  attributes  are  present,  (c.g.  a floating  point  number)  This  is  application 
dependent.  One  may  recode  a many-values  attribute  to  reduce  its  range,  which  has 
the  advantage  of  being  user  dependent  and  not  class  dependent.  A class  dependent 
partition  as  implemented  by  SL1Q  could  not  satisfy  the  user  point  of  view.  Another 
approach  is  to  prune  the  final  tree  and  merge  branches  that  lead  to  the  same  class. 


The  following  is  a list  of  the  SLIQ  features  versus  a decision  tree  approach  as  proposed 
in  this  work: 


L SLIQ  derives  a classifier  (a  decision  tree)  equivalent  to  other  ontropy  based  algorithms 
(C4.5).  It  uses  the  gini  index  as  the  criterion  for  attribute  selection. 


index  is  used  despite  the  performance  differences,  and  so  the  accuracy  as  a classifier 


1 have  already  implemented  a splitting  approach  (range  compression  algorithms) 
subtrees  in  disk,  which  makes  the  system  scalable  in  the  same  sense. 


3.  SLIQ  requires  at  most  one  pass  over  the  data  (two  times  the  number  of  I/O ’s  due  to 


of  the  decision  tree.  It  assumes  the  class  list  can  be  kept  memory  resident. 


Our  tree  derivation,  which  is  essentially  the  SUQ  system,  takes  at  most  one  pass  per 
level  or  the  decision  tree  (one  time  the  number  of  I/O’s)  if  we  keep  the  last  level  class 
counts  memory  resident;  which  is  the  same  assumption  that  SI, IQ  makes  about  the 

4.  Rules  extracted  from  a decision  tree  derived  from  SUQ  will  be  based  on  the  binary 
splits  of  the  attributes  ( /t  < n or  A > e)  and  hence  they  will  be  longer,  more  general 
and  less  meaningful  than  rules  based  on  a decision  tree  based  on  interval  splitting  for 
attributes  (our  system). 

5.  SUQ  requires  two  times  more  space  than  the  original  database  (since  columns  are 
kept  as  separated  list  with  indices  attached)  while  the  basic  decision  tree  algorithm 
requires  at  most  one  time  more  space  which  can  be  reduced  to  a minimum  using  an 


d.  Meanwhile,  our  decision 

7.1.3  Memory  Comparison 

The  following  assumptions  have  been  made  for  comparing  both  systems  on 
1.  Both  systems  will  be  used  for  tree  derivation  but  not  for  incremental  purpe 


2.  SLIQ  will  keep  the  Class  List  in  main  memory 
class  counts  of  the  tree  or  trees  in  memory  ami  only  those  since  we  are  concerned 

3.  We  use  the  gini  index  as  the  method  to  select  the  best  attribute  (to  make  the  com- 

4.  The  database  consists  only  of  numerical  features.  This  can  be  relaxed  if  we  modify 

Thus.  SLIQ  will  require  N • 8 bytes  of  main  memory  where  ,Y  is  the  number  of  tuples 
of  the  database.  If  A is  the  number  of  attributes.  V the  average  number  of  values,  C the 
number  of  classes  and  /f  the  height  of  the  decision  tree,  then  we  will  have 

2»M-//),K.C.16  (7.1) 

bytes  of  memory  required  for  our  algorithm  (Ihel'IVLD  algorithm).  Note  that  each  counter 
requires  16  bytes,  for  keeping  track  of  the  attribute,  value  and  class.  There  are  {A-H  )*V*C 

Then,  SLIQ  will  require  more  memory  if 

2"+\A-H),V*C<N  (7.2) 

tnents.  Note  that  the  height  II  is  unknou'n  before  hand  and  it  must  he  estimated.  If  the 
highest  value  for  height  A is  chosen,  then  we  might  use  SLIQ,  when  in  practice  a TIVLD 
(Tree  Induction  for  Very  Large  Databases)  algorithm  will  perform  better  if  the  decision 


tree  is  small.  In  any  case,  SMQ  will  require  two  times  the  number  of  I/O’s  anil  an  initial 
presorting  phase. 

7.1.4  Conclusions 

The  results  obtained  by  Mehta  ct  al.  [26]  show  that  SLIQ  can  be  used  effectively  for 
large  data  sets  with  linear  scalability.  The  comparisons  shown  in  that  paper  with  other 
systems  seem  unfair,  since  they  were  designed  with  different  goals  in  mind:  to  keep  data 
tuples  inside  the  tree  and  minimize  the  amount  of  central  memory  required,  without  caring 
about  the  number  of  passes  over  the  data  and  so  on. 

The  theoretical  comparison  made  here  shows  that  while  keeping  the  same  goals,  the 


to  the  values  that  each  attribute  may  have  (as  long  as  they  are  normalized,  at  least  in  INF), 
transaction  consist  of  a collection  of  items  (I). 


that  two  subsets  of  items  X,  Y are  associated  if  there  are  transactions  that  contain  both  X 
and  Y.  In  addition,  it  is  assumed  that  an  implication  of  some  sort  exist  from  X to  Y and 


is  denoted  X » Y.  The  support  and  confidence  were  defined  in  terms  of  the  number  of 
transactions  with  X , Y and  A'  U Y.  The  support  in  this  sense  is  the  maximum  number  of 
transactions  containing  X or  containing  Y.  The  confidence  is  the  ratio  j ^ - Note 

that  our  definition  of  support  of  the  association  rule  is  the  support  of  the  antecedent. 
According  to  Agrawai  [1],  the  support  of  a rule  is  constant  and  doesn't  depend  on  the 
support  of  the  antecedent. 

mapped  to  a row,  where  there  will  be  1 if  the  item  described  in  the  respective  column  is 

Thus,  if  an  association  rule  exists,  in  the  sense  of  Agrawai  in  [1],  between  two  sets  X 
and  Y,  then  let  S be  the  set  where  X=1  (i.e.,  where  every  item  column  is  1 ) and  C be  the 
sot  where  both  X=l  and  Y=l.  Thus,  we  have  X •—  Y ( s,  c)  with  s the  cardinality  of  S 


algorithms  to  association  rules  ones. 

The  translation  is  easy.  Consider  an  “item"  every  value  of  every  attribute.  Thus,  each 
tuple  will  be  mapped  to  a “transaction"  that  contains  all  values  of  all  attributes  in  the 
tuple.  If  a value  is  missing,  simply  don't  include  the  "item".  Note  that  the  sine  of  the 


7.2.2  Global  Features  Comparison 


a comparison  between  the  decision  tree  approaches  (DT)  and  association  rules  approaches 

(AR). 

the  data  base  with  a minimum  specified  support  and  a minimum  specified  confidence. 
The  consequent  of  the  need  not  be  simple  as  it  can  be  a composition  of  several  items 
(or  attributes). 

since  the  goal  attribute  has  several  values  and  each  value  is  an  "item"  in  the  items 
database,  the  DT  algorithms  are  extracting  several  rules  simultaneously. 
Additionally,  if  A *-•  fl(sl.cl)  and  A ” C(s2,c2)  then 
A *—  B + C,(max(sl,*2),min(cl,c2))  + denotes  concatenation. 

In  this  case,  only  single  consequent  parts  are  necessary;  this  diminishes  the  amount 
of  parallelism  needed. 


• Redundant  work:  On  the  other  hand,  AR  algorithms 
above  whenever  the  first  two  rules  satisfy  the  thresholds  required;  creating  some 

"item"  is  related  to  the  same  attribute  in  the  original  data  base. 


Range  compression  and  recodtficalton:  It  is  easy  to  incorporate  range  compression 
to  DT  approaches  which  allow  us  to  create  rules  with  larger  support.  It  is  possible 

creating  the  items  database  but  range  compression  is  just  a feature  of  DT  approaches 
and  can  not  be  applied  with  an  AR  approach. 


are  criteria  that  can  be  applied 
rules  which  are  not  present  in  a 


simple  standard  to  item  transformation. 


This  information  is  lost  with  the  transformation  to  an  items  database . 


• Incremental  approaches'.  There  have  not  been  proposals  to  implement  an  incremental 
approach  to  extract  association  rules.  In  all  AR  algorithms  the  whole  database  is 
processed.  However,  A,  Savascrc  minimizes  the  number  of  passes  over  the  database 

Pol- 


and is  interested  only  the  simple  rule  extraction,  I present  a theoretical  comparison  of  the 
DT  algorithms  with  the  actual  AR  algorithms  below. 


In  order  to  use  decision  trees  we  have  to  define  target  attributes.  Since  we  don't  know 
beforehand  which  attributes  are  important  for  the  application,  a general  approach  is  to 


derive  the  decision  tree  for  every  attribute.  (In  a real  application,  people  will  be  interested 


in  specific  attributes,  except  if  they  want  every  possible  relationship).  Then,  we  have  A 
decision  tree  derivations  , where  A is  the  number  of  attributes.  In  order  to  get  the  same 

decision  tree  approach  will  be:  CDT=  A ’(Passes  to  build  the  tree  + one  pass)  since  we 

attributes  (there  will  be  A passes  over  a subset  of  the  transaction  data  base)  So,  Passes  to 
build  the  tree=  A*S  where  S is  the  proportional  sise  (0  < S < 1)  of  any  subset  of  the  data 

base  and  CDT  = A(A  • S + 1)  = A + S * AJ. 

In  general,  this  precludes  the  use  of  a classical  decision  tree  algorithm,  because  the 
association  rules  algorithm  will  make  at  most  A passes  over  the  data  and  CDT  will  bo 


The  MGDT  algorithm  described  in  chapter  3 derives  the  decision  tree  for  a set  or  all 
attributes  in  the  database.  It  derives  the  trees  breadth  first  (different  from  our  Revisited 
Derivation  algorithms)  and  reads  the  data  base  once  at  each  level  of  all  trees.  Thus,  we 

This  is  a comparison  of  the  association  rules  system  (Apriori  algorithm)  (AR)  and  our 
approach(MGTD): 

over  the  database  is:  CAR  = 0(A)  and  the  complexity  of  the  MGTD  system  is  CDT  = 
0(A  * S + 1),  where  0 < S <=  1 if  the  A decision  trees  are  derived  in  parallel. 


In  this  case,  the  decision  tree  approach  will  be  belter  in  general  if  5 < 1 - 1/A,  which 
will  be  true  almost  for  every  A. 

cases,  with  the  same  complexity  CDT  = 0(A)  since  5=1,  and  the  test  phase  is  not 


The  MGDT  algorithm  offers  an  additional  advantage:  it  is  possible  to  speed  up  the 
itemset  and  the  user  confidence.  It  could  be  the  case  that  none  of  the  subsets  satisfy  the 


user  confidence  for  the  rule.  It  is  not  possible  to  use  the  user  confidence  before  the  whole 


CHAPTER  8 

CONCLUSION  AND  FUTURE  WORK 


The  decision-tree  approach  is  important  since  decision  trees  solve  the  classification  prob- 
lem and  I have  shown  that  can  be  effectively  used  for  rule  extraction.  Then  their  application 
to  Knowledge  Discovery  is  imminent.  Their  application  to  very  large  data  bases  (distributed 

preserving  the  accuracy  of  classification  and  the  confidence/support  of  the  potential  rules. 
Ours  is  one  of  the  first  attempts  to  propose  and  use  decision  trees  for  discovering  quantita- 

in  Knowledge  Discovery  and  Data  Mining.  See  for  example  [43],  [19]. 

that  must  be  enhanced  in  the  model  are  mentioned  if  one  likes  to  use  our  approach  of 


developed  for  a potential  large  database  for  ECMO  (ExtraCorporea!  Membrane  Ox- 
igenatlon)  data.  This  database  was  created  by  Drummond  [11]  and  it  consists  of  a 
large  clinical  data  collected  minute-by-minute  of  infants  who  are  critically  ill.  The 
data  was  reduced  for  the  purposes  of  the  application  to  a small  data  set  of  relevant 


cases  and  the  results  are  not  included  here  since  most  of  the  features  proposed  in  this 
work  were  not  applicable;  however  the  results  were  meaningful  for  the  expert. 

The  use  of  decision  trees  in  the  experiments  and  in  the  practical  application  allows  us 
to  visualize  several  operators  that  must  be  implemented  in  a Data  Mining  Manipula- 


menting  the  data  base  is  not  useful  in  this  case  since  we  have  to  locate  each 

pointers  to  the  original  database.  The  DMML  should  provide  this  capability  to 
the  Decision  'l>ee  Based  System. 


concern  for  the  end-user.  Developing  the  decision  trees  or  rules  for  all  of  them 
is  not  required  or  important.  The  DMML  must  be  able  to  provide  only  the 

can  achieve  this  requirement  but  still  this  is  not  transparent  enough  for  the  Data 

- Attribute  joining:  Decision  Tree  algorithms  can  be  complicated  enough  when 
only  a goal  attribute  is  used.  If  several  attributes  must  be  considered  as  a 
unique  goal  attribute:  the  algorithm  does  not  change  but  the  interface  must 
suffer  a lot  of  changes.  The  DMML  must  provide  a way  to  retrieve  as  a unique 


can  be  captured  only  in  aggregate  attributes.  A way  to  combine  and  summarize 


must  be  provided. 

- Irrelevant  Attributes:  When  there  are  attributes  that  are  not  required  to  be 
processed,  the  DMML  must  help  in  this  matter.  Although,  SQL  statements  are 


- Keep  primary  keys:  For  data  analysis  and  classification,  the  user  might  need  to 

- Pre-selected  attributes:  In  the  same  way  that  some  attributes  can  be  considered 

- Frequency  calculations:  Most  of  the  decision  tree  derivation  time  is  time  ex- 
pended in  frequency  calculations.  If  the  Database  System  has  effective  and  efli- 


when  they  are  read-  to  lower  the  number  of  classes  when  they  are  used  as  targ 
attributes.  Thus  no  real  changes  are  made  to  the  database. 


Focus  component:  Selection  statements  in  the  interface  as  well  as  keeping/  discarding 
attributes  are  ways  to  focus  in  the  necessary  data.  Additionally,  in  the  early  stages 
of  decision  tree  derivation,  attributes  with  low  certainty  measures  can  be  discarded 


from  additional  computations.  Minimum  threshold  values  can  be  provided  to  do  this. 


criteria,  the  range  compression  algorithms,  the  distributed  algorithms,  incremental 
approaches,  and  induction  in  large  databases. 


lecision  trees  allows  us  to  extend  them  to  the  most  complicate  types  of  rules  which 
re  of  actual  interest.  See  [42]  , [20]. 


"When  faced  with  a high-dimensional  attribute  space,  tree-based  techniques 
which  in  a greedy  fashion  split  the  data  one  attribute  at  a time  are  generally  far 
superior  to  techniques  which  require  examining  some  combination  of  attributes". 


A number  of  issues  that  are  not  fully  addressed  in  this  work  arc: 
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Full  implementation  of  the  system-  The  implementation  I made  for  experimental  pur- 


Analysis  of  the  effects  of  the  incremental  approach  with  respect  of  the  shape  of  the 
decision  tree  and  in  the  final  rules.  It  is  dear  that  the  previous  trees  resemble  the 


this  resemblance  in  terms  of  the  number  of  matching  nodes,  branches,  height  and  so 
on.  Experiments  with  large  data  bases  are  important  for  this  purpose. 


periments,  but  the  behavior  of  the  decision  trees  algorithms  in  most  uncontrolled 


Use  of  DMML  The  Data  Mining  field  is  just  emerging.  Researchers  are  doing  mostly 
file  mining  rather  than  database  mining  [22].  When  DMML  are  available,  it  will  be 
important  to  evaluate  the  performance  of  decision  tree  algorithms.  See  [21], 
Improving  on  the  range  compression  algorithm  The  implementation  of  the  range  com- 
pression does  not  include  the  Best  Split  algorithm  described  in  chapter  fi.  It  seems 
a Look-up  algorithm  can  be  easily  implemented  since  most  subrange  calculations  are 
repeated  in  the  algorithm. 


Incorporate  a way  to  make  the  selection  criteria  user  dependent.  Although,  we  have 
incorporated  several  criteria  into  the  implementation,  new  criteria  will  require  new 
programming.  User  dependent  implementation  of  criteria  can  be  better  suited  to 
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